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THE GRE*Q COACHING/INSTRUCTION STUDY 



Until recently» the conclusions of research studies designed 
to test whether some type of shorf-term instruction^ could produce 
meaningful gains on standardized aptitude test scores have been 
uniformly negative « In these studies* several of vhich have been 
conducted by or for the College Entrance Examination Board (CEEB) 
since 1950» a nuud^er of problems are evident* These problems are 
based upon the subjects* level of ability and test sophistication* 
the amount of instruction provided* and the quality of the instruc- 
tional materials and of the instruction itself* In most studies* 
the subjects tended to be academically very able and highly 
sophisticated in test-staking strategies* and they were therefore 
not likely to benefit from, short*term instruction* A more recent 
study (Roberts S Oppenheim* 1966)* concerned with disadvantaged 
students* indicated that their subjects were probably too deficient 
in their general academic preparation to benefit from short-term 
instruction for an SAT*level test* In most of the studies that 
have been conducted* instruction was too brief (from 7 to 15 hours 
at most) and too diverse (including both verbal and mathematics 
content) to test the effectiveness of short-term instruction* 
Finally, the instruction was typically unstructured* no attempt 
having been made to develop a systematic set of instructional 
materials to be administered by well-^crained ^d properly oriented 
teachers* 

The question of the susceptibility of the SAT to short-term 
intervention arose again when it was proposed that a new type of 
mathematics aptitude ±tem (Quantitative Comparison* or QC) replace 
the more traditional (Regular Math* or RM) and the Data Sufficiency 
(DS) items currently found in the SAT-M* The new item format had 
the demonstrated advantages of providing greater reliability and 
apparently equivalent validity for a given unit of testing time* 
but there was a concern about the susceptibility of the new item 
type to short-term intervention* 



Throughout this report* the terms short-term instruction 
and short-term Intervention are used interchangeably to mean a 
structured program including instruction for content as well as 
test-taking strategies* Such instruction* if successful* would 
probably result in changes in criterion performance as well as 
test performance* Coaching is traditionally defined as instruc- 
tion in test-taking strategies and/or in the, peculiarities of a 
particular test* Coaching* if successful* would probably result 
in changes in test performance* but not in criterion performance* 
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At the request of the CE£B» Evans and Pike (1973) undertook 
a study to determine the relative susceptibility of the three 
mathematics item formats (QC» DS» and HK) to score changes 
resulting from short-term intervention. The main emphasis of 
the SAT-M study was to determine whether the QC item format was 
susceptible to some sort of short-term intervention and» if &o» 
to compare its susceptibility with that of the RM and DS formats. 

In general* the results of the Evans and Pike study confirmed 
the hypothesis that each of the three item formats was at least 
somewhat susceptible to instruction. The study clearly established 
that if highly motivated high school students are given well'- 
prepared and well-presented short-term instruction in appropriate 
mathematics content and test strategies, significant gains in 
mathematics aptitude test scores are likely to result. 

T 

Given the success of this study* it seemed likely that similar 
results could be obtained with subjects taking aptitude tests at 
a higher level — that is» candidates for the Graduate Record Exami- 
nations. The GRZ-Q is similar to the SAT-M in general content and 
difficulty level. Neither Cest assumes that candidates have 
received instruction in mathematics beyond high school algebra and 
geometry* and many of the candidates are neither highly sophisti- 
cated nor necessarily '*fresh'* in their iinderstanding of basic 
mathematics concepts. There are several differences* however4 
SAT-M at that time used two item formats / Regular Math and Data 
Sufficiency* with a fairly large percentage of geometry items* 
The GRE-Q does not use Data Sufficiency ite^s^^ut does* on the 
other hand* use the Data Interpretation format that requires the 
candidate to answer questions using data presented in tabular or 
graphical form* It also contains very few geometry items and has 
many more word problems than the SAT-M. 

The purposes of the present study were to determine the 
susceptibility of the GRZ-Q to short-term instruction and to assess 
the differential effectiveness of this instruction for black* 
white, and Chicano ORE candidates. The study included four phases: 
(1) the feasibility study* (2) the exploratory study, (3) the 
developmental phase* and (4) the operational phase. 



The Feasibility Study 

The feasibility study was a survey of several colleges similar 
to those that would be asked to participate in the final opera- 
tional phase of the study* The purpose of t;his survey was to 
determine the extent of students* interest in and need, for a 
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program of special IntenAlve instruction for the mathematical 
part of the Graduate Record Examination, and to determine the 
degree of interest and cooperation of the faculty and adminis- 
tration in the program* 

fhe Sample 

Colleges , Twelve colleges, chosen because of their geographic 
location and the nature of their student bodies, were invited to 
participate in the feasibility study. Two geographic areas were 
identified for this sample: the Southeast/Southwest region and the 
Northeast/Midwest/Far West region* A distribution in the range of 
colleges according to the nature of their student bodies was sought 
on the basis of racial or ethnic composition: integrated (multi- 
racial) or predominantly black, Chicano, or white* Two of the 12 
schools (both integrated) originally invited declined to participate 
in the survey. Of the 10 remaining schools, four were predominantly 
black, two predominantly Chicano, two predominantly white, and two 
integrated. 

S tudent s. Because of time pressures, the selection of students 
to be surve/ed was left to the discretion of the college administra- 
tcr who conducted the survey on each campus. Some colleges chose 
to survey a one-half or one-third sample of tfaeit junior classes. 
In these colleges, a questionnaire was mailed to each student chosen 
for the survey, and the student returned the completed questionnaire 
directly to ETS. In other colleges where juniors were surveyed in 
the classroom, the questionnaires were collected in the classroom 
and returned to ETS in one batch. The principal investigator con- 
ducted the survey on two of the campuses. On these campuses admin- 
istration of the survey was preceded by a lecture about the ORE and 
its various items. Although the lectures were advertised and were 
scheduled at a time when they would not conflict with other campus 
activities, on both campuses conflicts with other Important activ- 
ities did arise and probably accounted for the somewhat smaller than 
expected attendance. At each of these campuses additional responses 
to the survey questionnaire were solicited through classroom instruc- 
tors* 

Questionnaire 

The survey material included a one-page description of the 
program of instruction and a one-page questionnaire* The descrip<- 
tion of the program stressed the reasons for conducting a special 
intensive Instructional course for the mathematical portion of the 
GRE, the nature and content of the course and of the instructional 
materials, the amount of time involved in classroom instruction 
and in out-of-class work, and the fact that the course would be 
provided at no charge to the student* The questionnaire requested 
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the student to indicate the extent to which he or she would participate 
in a program such as the one described* and the amount of time per 
week he or she would be willing to devote to the course* and also 
asked for information on the student^s undergraduate major» intended 
field of graduate study* and ethnic background. 

Results 

The results of the feasibility survey for five colleges (tvo 
predominantly Chicano» one predominantly white* one predominantly 
blacky and one integrated) are suimnarized in Table 1- It is 
unfortunate that many schools were in the midst cf final examinations 
at the time the survey was conducted* and so there was a much lower 
response rate than had been anticipated. Some schools declined 
because of end-of-^the-year time pressures* and a few schools did not 
provide usable results. 

The data presented, in Table 1 were prepared for an interim report 
to the GR£ Research Committee* for which time data from only five schools 
vere available. The decision to continue the study was based on these 
data. 

The response to the questionnaire was overwhelmingly positive. 
Students indicated that if a special instruction program for GRE-Q 
were offered on their campus* they would volunteer* and in addition 
showed a willingness to devote a substantial amount of time to such 
a program. The najority of the respondents were willing to devote 
at least two"^ classroom hours per week to the course. All administra-^ 
tors surveyed indicated keen interest in the project and a willingness 
to cooperate- Discussions with administrators at colleges in the 
survey that had not yet provided data seemed to support the results 
shown In Table 1. On the basis of the evidence that sufficient 
student and administrator interest did exist in the proposed programs* 
a recommendation to continue the developmental efforts was made to the 
GREB Research Committee* and the Committee concurred with the recom-^ 
mendation. 



The Exploratory Study 



The exploratory study was designed to pro\'ide the principal 
investigator and the test development staff with specific InfonDa-^ 
tion about the problems that students encounter when responding 
to GRE-Q items* thereby identifying the weakness of students and 
the areas where intensive instruction was needed. This information 
then served as a basis for the '^svelopment of the experimental curric- 
ulum. The principal investigator and the test development staff 
member who was to be involved in developing the instructional materials 



Table 1 

Summary of Results of Student Survey at Five Colleges 
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Other ethnic 
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Visit- 
Classroom 
Survey 


85 


4 


8 


2 


5 
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0 


1 
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B 


Predominantly 
Uhite 


SE/SU 


Mail 


6 


0 


0 


D 


191 


64 


4 


0 


1 
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Predominantly 
Chicano 


se/su 


Visit*' 
Classroom 
Survey 


5 


1 


98 


27 


64 


16 


0 


X) 


0 


0 


D 


Predominantly 
Chicane 


se/su 


Mail 


1 


0 


16 


1 


3 


3 


4 


0 


0 


0 


E 


Integrated 


ne/mw/fw 


Mail 


31 


3 


2 


0 


198 


54 


12 


2 


2 


2 
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8 
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1 1 

20 
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2 


4 
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Not blacky Chicano^ or white 
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visited three campuses (one predominantly black* one predominantly 
Chicanot and one multiracial) and tested and interviewed a total 
of 44 student volunteers. The volunteers were assembled in a group 
on each campus and given one of two forms of a 20-minute multiple- 
choice test consisting of 15 items taken from the GRZ-Q item pool. 
After the test» the research team held a 30-minute interview with 
each volunteer. During this interview* the student was asked to 
reconstruct the reasoning process by which he or she had arrived 
at an answer for each problem on the test* The interviews were 
taped* and extensive notes were taken* Subsequently* each of the 
30 test items was categorized according to the specific mathe- 
matical concepts considered necessary to answer it. 

Responses to the items were then categorized according to the 
difficulty that the various concepts presented to the students. 
Prom this information* an outline of mathematics concepts that 
needed to be covered in an intensive instructional program was 
developed* That outline is presented in Table 2* The research 
staff and the test development staff refined the outline and de- 
vised plans to develop materials (teachers^ outlines and student 
workbooks) to cover each of the major topics of the outline* The 
original plan of the study called for six classroom sessions to 
cover the specified mathematical concepts* The GREB Research 
Committee requested* however* that an attempt be made to test the 
effectiveness of a two-hour test familiarization session* Because 
of the additional time necessary for that session and for post- 
testing to messure i£s ef;fectiveness* the mathematics concepts 
were covered in four classroom periods rather than six* 



The Developmental Phase 



The coaching/ instruction curriculum materials that were developed 
required eight sessions of approximately two hours each* the first 
and last of which were devoted to testing* The second session was 
devoted to test familiarization aimed at alleviating the candidate's 
anxiety about the admissions process* Por this purpose there was a 
presentation of a practice with various general testing strategies — 
e.g** pacing and when and how to guess* Further information included 
the uses of tests in the admissions process* the legitimacy of admissions 
tests (validity and reliability)* the GR£ test directions* and the 
answer sheets. 

Thti third session consisted of a short (40-item) mathematics 
test designed as a scaled-down version of the GRE-Q with similar 
items and content. The purpose of this session was to assess the 
effects of the second session* and the test was reviewed item by 
item as a learning exercise. 
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Table 2 
Content Outline 
Outline for GRE Math Coaching Study 

Lesson 1 
Coinputatlonal Skllla 
A« Whole numbers 

1* Addition and multiplication with carrying 

2* Subtraction with borrowing 

3* Division 

a« With and without remainders 

b* With powers of 10 (lOtOOOt not 10 ) 

c« Zeroa In divisor and dividend (ex« 305) 

B« Properties of numbers 

1* Odds — evens 

2* Factors and divisibility 

3* Prime numbers 

A* Factorisation into primes 

5* Least common multiple ^ 

6* Greatest common divisor 

C* Fractions 

1« Reducing and equivalent fractions 
2* Multiplication 
3« Reciprocals 
4 * Division 

Addition and subtraction of unlike fractions 

Decimal fractions ^ 

1* Decimal notation-powers of 10 (not 10 ) 
2« Equivalent decimal fractions 
3* Addition of decimal fractions 
4« Subtraction of decimal fractions 
Multiplication of decimal fractions 
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Decimal fractions (continued) 

6* Divlaion by whole numbers 

7* Division by decimal fractions 

8* Multiplication by powers of 10 

9* Division by powers of 10 

Leaaon 2 

A* Ratio^proportion 

Percent 
C« Averages 

Lesson 3 

Algebra and geometry^ measurement 

A« Signed numbers and number lines 

B* EvaluatiJDn of algebraic expressions 

C* Solution of open sentences 

X* Equalities and inequalities in 1 unknown 
2« 2 equations in 2 unknowns 
3* Verbal problems 

a* Setting up 

b* End-points 

D* Exponents and roots 

1* Basic laws of exponents 

2* Square roots (emphasis on aquaring» not taking roots) 
3. Expanded notation 

E« Operations with algebraic fractions 

Area and perimeter of triangle* parallelogram, rectangle* 
and square* and the diagonal of the square 

G* Area and circumference of a circle 

H* Volume of rectangular solid* cube* cylinder* and sphere 
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Algebra and eeouetry^ measurement (continued) 
I« Ifl08cel«8 and equilateral triangles 

J* Pythagorean theoremt right triangles, and special triangles 
(3-4-5. 30(1) - 60(2)- 90(0^7), 45(1) -45(1) *90 (VD) 

Relation betveen aize of sides and angles of a triangle 

Li Angle measurement 

1« Acute, right, obtuse, straight 

2* Vertical 

3* Sum of angled of triangle and quadrilateral 

4* Angle veasurement related to parallel lines 

Coordinate geometry . [rectangular coordinate system] 

Leaaon 4 
Data interpretation 
A« Bar graphs 
B. Line graphs 
C* Circle graphs 
D« Tables and charts 

E. Dnusual grapha [triangular, flowcharts, etc*] 
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Seaaiona 4 through 7 covered Che basic conCenC ChaC had been 
ldeat;i£ied in the exploratory study as neceasary for adequate per* 
fonaance on the GR£*Q« These aeasions and Che content of each are 
Itated below; ' 

Session 4 - basic number facta, operations with fractions, 

lowest comnon iDultiple, greatest comon divisor* 

Suasion S - averages, ratio and proportion, and percent* 

Seasion 6 - basic linear algebra, geometry* 

Seaaion 7 - data interpretation/ 

The curriculum materials were developed mainly by the mathe- 
marlca ataff of the ETS Test Development Division, using standard 
procedures for item'^vriting and 'for thft preparation of the testa 
and for the vorkbooks, class books, mini*lessona , and lesson plana* 
Detailed explanations of most problems included in the materials 
allowed the student to follow the solutions step*by*atep* 

The materials included leason plans for all sessions for the 
teachers, and vorkbooka, class books, and mini*lessona for Sesaiona 
4 through 7 for the atudents* The lesson plans for the three testing 
sessions (Sessions 1, 3, and 8) gave instructions on how to administer 
the tests* The lesson plan for the test familiarization seasi'jn 
(Session 2) presented ideas for discussion as well aa sample GRE 
questions* The outline of the remaining £our lesson plans covered a 
diacuaaion of the mini^lesson with examples for the claas to try, a 
discuasion of the topics to be covered in that class (again with 
examplea), and keya and hinta for the claaaroom exercises, classroom 
test, and diagnostic teat* 

The workbooks conta^Vied exercisea dealing with the concepts 
taught In the mini-lesaona. The format was one of inmtediate 
feedback: the right-^hand page contained several problems and 
apace in which the atudent could work the problems; the reverse side 
of the page gave explanationa of the solution of the problems and 
the correct answers* The last exercise in each lesson was a timed 
practice test, with answers and explanationa at the end* Students 
were expected to do the workbook exercises on their own time and to bring 
up *any problems or questions regarding them in the next class t 

The claas books conaisted of a few practice exercises » S or 10 
minutea long, to be completed by the atudenta in class ^d then 
discussed* Each class book also contained a class test and a short 
diagnostic test* The class test consisted of 15 problems to be solved 
in 20 minutes* It was suggested that the instructor discuss the class 
test problems and their solutions in class^ The diagnostic test, to be 
taken in class, was turned in to the instructor for marking after class* 
The results of the diagnostic test were to be used by the instructor 
to determine particular strengths and weaknesses of the class. 
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The mini^leflsons were one- or two^p^ge sunnnaries of che major 
points to be covered in the lessons. They were designed for students 
to use for review and aa reference material* The student was en- 
couraged to memorize the concepts presented in the mini-lessons* 



The Operational Phase 



The next phaae of the study was deaigned to test the effective- 
ness of the course of instruction described above as a m^ans of 
increasing the GRE^ scores of candidates* especially minority 
candidatea. In order to accomplish this objective* a special 
course was offered to volunteer GRE candidates on 12 campuses. 

Procedures 

Schools * The schools invited to participate in the project 
were selected according to the makeup of the atudent body» school 
control (state or private)* and geographical location* The schools 
included four predominantly black schools in che southeaat and 
southwest United States* one predominantly Chicano school in the 
southwest* two predominantly white schools in the southeast* three 
multiracial white-black schools in the midwest and northeast* and 
two multiracial white^Chicano schools in the southwest* Some of 
the schools were selected on the basis of the results of the 
feasibility study* and some were selected because of their expressed 
Interest in the project* 

Coordinators . An attempt was made to select on each campus 
a project coordinator who was highly viflible and was respected by 
the target groups. For example* on one campus* the project 
coordinator was a Chicano man working in the office of Special 
Projects and dealing directly with the academic problems of 
minority students. On other campuses* coordinators were connected 
with academic foundations departments (minorities special studies 
programs), testing offices* and psychology departments* A 
coordinator's main role was as a liaison between the project staff 
in Princeton* the instructors* and the administration of the 
university* Specifically* coordinators were asked to suggest 
potential instructors for the courses*^ to make arrangements for 
epace for testing and Instruction* and to recruit atudent volunteers 
for the project. 

Teachers . Teachers were recruited from the faculties of nearby high 
schools or from college instructors in special remedial programs. The 
decision not to use college mathematics professors was based on the 
belief that such persons would not be extremely interested in or 
effective at teaching mathematics at this level and of a remedial 
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nature. Of the 17 teachers engaged for the project, eight vere 
currently teaching mathematics at the high school level* and nine 
were remedial education specialists at the college level* A one* 
day workahop was held in August 1973t for the purpose of familiar- 
izing the teachers with the materials and procedures of the project* 
Sixteen of the 17 teachers attended. 

In July 1973f instructions for recruiting and registering 
student volunteers for the course were sent to the project 
coordinator* The instructions outlined criteria for student 
participation and suggested ways to go about recruiting and 
registering stu<l^nt8* the principal criteria for student partici- 
pation in the project vsre that students should be seniors, should 
be thinking of attending graduate school and therefore considering 
taking the GR£ aptitude tests, and should not be mathematics or 
*^ard science" (^e,g*» physics) majors* The coordinators were 
instructed to inform students about the experimental nature of the 
project and the amount of time and work» as well as testing* that would 
be required* The actual procedures used tp recruit students were left 
to the discretion of the coordinators* They were provided* however, 
with sample press releases and a sample announcement to mail to each 
student selected for the project* They also had an outline of the course 
and detailed instructions for registering the students* 

The number of student volunteers to be recruited on each campus 
depended upon the size and type of school* The seven predominantly 
uniracial schools had one class for each of the experimental and 
control groups; the five multiracial schools had two classes for 
each of the experimental and control groups— a total of 17 classes. 
The maximum enrollment of each class was 20 students* 10 male and 
10 female* On the predominantly uniracial campuses* coordinators 
were asked to recruit 40 subjects* 20 male and 20 female* Half 
of each of these groups were to be assigned at rsndom to either 
an experimental or a control group* On the multiracial campuses* 
coordinators were asked to recruit 80 volunteer subjects* 40 
minority and 40 nonminority* half female and half male* from which 
the experimental and control groups would be assigned randomly. 

Results 

Table 3 shows the number of students called for by the original 
plan on each of the 12 campuses* Table 4 is a diagram of the research 
design for subjects in each school* The two experimental and two con- 
trol groups represent random halves of these groups resoectivelv* 
Table 5 shows the nunoiber of students actually recruited on each of the 
campuses— that is* the number of students who volunteered for the f<roJ-- 
ect and who appeared for the initial session* The last column of / 
Table 5 (Complete Data) shows the total number of subjects in eai^H 
school (without regard to sex or ethnic group) from each of whom a 
usable pretest and posttest were obtained. 
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Table 3 

Number and Type of Students to Be Recruited 
from Each of 12 Schools 



School 
tluober 


School Type and 
Control* 


tlumber of 
Classes 


Black 
M F 


Whlce 
M F 


Chlcano 
M F 


Total 
M F 


01 


Black (Pr.) 


1 


20 


20 










20 


20 


02 


Black (St.) 


1 


20 


20 










20 


20 


03 


Chlcano-Mhlte (St.) 


2 






20 




20 


20 


40 


40 


04 


Black*Whlte (Sc.) 


2 


20 


20 






20 


20 


40 


40 


05 


Chlcano (St.) 


1 


- 


- 






20 


20 


20 


20 


06 


Chlcano-Whlte (Sc.) 


2 






20 


20 


20 


20 


40 


40 


07 


Black-Uhlce (Sc.) 


2 


20 


20 


20 


20 






40 


40 


08 


Uhlce (Sc.) 


1 






20 


20 






20 


20 


09 


Black (Pr.) 


1 


20 


20 










20 


20 


10 


Whlce (Pr.) 


1 






20 


20 






20 


20 


11 


Black (Sc.) 


1 


20 


20 










20 


20 


12 


Black-Whlce (Sc.) 


2 


20 


20 


20 


20 






40 


40 


locals 


17 


140 
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Diagram of Experimental Design for Sub,1ects in Any' School 
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Table 5 



Number of Subjects Regiacered in Each Ethnic and Treacnenc Group» Total Pocenclal 
SubJecCfif and Subjects Who Supplied Proper Pre* and PoaCCeacs ac 12 Schoola 







Resiatered 










Black 


White 


Chlcano 


Potential 


Complete 


Scnoo^. 


Group 


Hale 


FemAle 


Total 


Hale 


Feniale 


Total 


Male 


Female 


Total 


Subjects 


Data 


01* 


E 


9 


9 


IS 














18 


4 




c 


5 


11 


16 














Id 


13 




E 




1 J 


25 


1 


9 


1 








26 


0 


02 


ft 

C 


12 


12 


24 


0 


0 


0 










0 




B 


1 


0 


1 


D 


13 


4 A 

19 


6 


0 


D 


26 


5 


03 




0 


0 


0 


11 


lA 


25 


2 


2 


4 


^9 


2o 




E 


A 
H 


J 


/ 


A 
*l 


b 


10 


0 


1 


1 


18 


10 


04 


ft 


0 


A 
*? 


A 


2 


10 


12 


0 


0 


0 


16 


11 




E 








2 


1 


3 


A 


7 


11 


14 


8 


05 


C 












A 


A 






10 


7 




E 


0 


0 


0 


9 


9 


16 


10 


8 


18 


36 . 


25 


06 


' C 


0 


■1 

A. 


■1 


q 


q 


A. U 




7 


19 


38 


29 




£ 


5 


6 


13 


8 


A 

9 


17 


2 


0 




32 


9 


07 


C 


2 


9 


11 


5 


7 


12 


0 


0 


0 


23 


18 


AO 


E 
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9 


16 








16 


7 


08 


C 
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12 


19 








19 


19 
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12 


17 














17 


10 


09 
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19 
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2 


16 
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11 
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11 


14 
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2 


16 
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12 
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3 
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(1) 


(1) 


(2) 


15 


5 
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2 


5 


7 


? 


6 
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ro.) 
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(«} 




1« 




' E 


42 


64 


106 


39 


54 


93 


22(1) 


18(1) 


40(2) 


241 


89 


Total 


C 


31 


67 


98 


40 


62 


102 


19(0) 


12(8) 


31(8) 


239 


174 



* Numbers in () in these columns represent Puerto Rican students* 
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It should be noted that two schools (2 and 11) provided no usable d^ta« 
When aex and ethnic Identity are Included as design variables, the cell 
sixes drop to zero In numerous Instances* Also ''complete data'^ refers 
only to students from whom the full pretest (Session 1) and the full 
posttest (Session 2— control group, and Session 8 — experimental group) 
were obtained* Any attempt to assess the effects of the brief test 
familiarization program would result In even smaller sample sizes* 
These data are at best minimal* 

Because of the spottiness of the data, the approach to the analysis 
was to use all of the data available to answer the following three 
questions*^ 

1* Ifos the overall program of Instruction helpful In Increasing 
GRE-Q scores of the subjects? Was It equally helpful for all 
ethnic groups and both sexes? 

2* Was the effect, If any, of the program due to the 4 sessions 
of mathematics Instruction alone? Is this answer the same 
for all ethnic groups and both sexes? 

3* Was the overall effect simply due to the anxiety reduction 
session alone? Is this answer the same for both sexes and 
each ethnic group? 

(^estlon 1 was Investigated by using data on all experimental group 
subjects who took both the pretest (PR) and the posttest (PT) * Questlon^2 
was Investigated by using data from all experimental group subjects on 
whom both PT and anxiety reduction . (AK) test scores were available* 
(^estlon 3 used data from all control group subjects on whom both >PR and 
AR tests were available* In each case the analysis used every subject 
(In the appropriate group) for whom the two scores In question were 
available* 

It was expected that the effectiveness of the program would be 
dependent upon the Instructor and other aspects of educational settings 
as well as upon the student* Therefore, data were examined for each In- 
dividual class* The underlying approach to the analysis, then, was to 
employ Intralndlvldual differences and to analyze the data separately 
for each class* The results of these analyses are presented below* 



The analyses presented below were suggested by and performed by 
Lincoln Moses and Suzanne Wong* A complete report of their analyses 
appears In Brown, B* et al*, Blostatlstrlcs Case Book Volume One, 
Stanford, California: Stanford University, 197d The author wishes to 
tHank them for their assistance* 
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Question 1 

Waa the overall elght*veek program helpful In lncrea8;lng the GRE-Q 
8coreQ of the sublecta ? 

The poattest (FT) minus the pretest (PR) differences for all experi- 
mttntal subjects in each of the 15 classes on whom data were available 
were computed* The results are presented in Table 6* Since the varia- 
tions in means at'yoss the instructors is large compared with those within 
instructors^ wl thin- ina true tor vatiance is disregarded as underestimating 
the real uncertainty* It can be seen from Tabid 6 that thsre is a small 
consistent positive difference (12 of 15 classes show PT-PR differences 
in the positive direction)* Confidence intervals are shown at the bottom 
of the table* Both sign test and ^-test confidence intervals are pre- 
sented* When all 15 classes are used, the 96*4% sign-test-based confidence^ 
interval shows a significant median effect due to instruction* but the 
95% t^confidence interval based on all 15 classea includes zero* However* 
when the class (14) with only one subject is dropped from the calculations, 
the 95% t;*'Conf idence interval excludes zero* These results indicate an 
overall positive effect, which ia due to the instructional program that in- 
cluded both anxiety reduction and mathematics instruction* 

Table 7 presents results related to the differential effects of 
the instructional program that are due to the sex of the subjects* PT 
and FR scores were available for both male and female subjects in 13 of 
the 17 classes* In 6 classes the differences favored females; in the 
other^ 7 the differencea favored male subjects* Thus, there is no basis 
on which to conclude that the instruction had a differential effect for 
either sex* 

Similarly, Table 8 presents results related to the differential 
effects of the instructional program resulting from the subjects' ethnic 
identities* There were only 5 classes in which Chicano;*vhite comparisons 
could be made* and in these the differences favored Chicanos in 2 classes 
and whites in 3 classes* Black-white comparisons could be made in 4 
classes with blacks shoving greater average gains in one class and lower 
in 3 clasaes* These data do not lend support to thi hypothesis that the 
overall instructional program wafi differentially effective, because of 
the ethnic backgrounds of the subjects* Furthermore^ the small numbers 
of classes available for evaluation would most probably obscure whatever 
effects might exist* 

Question 2 

Tfas the instructional effect due to the mathematics instruction 

only ? 

In Table 6 it was demonstrated that there was a significant effect 
that resulted from instruction* However^ the instruction consisted of 
2 parts, a one-session program designed to familiarize the subjects with 
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Table 6 

M«ans and Variances^ of Intrasubject Differences (PT-^PR) 
in Experimental Classes 





Class _ 


Mean 


Variance 


4 


1 


0.012581 


0.0018412 


5 


f 

L3 


-0.010213 


0.011344 


7 




0.056568 


0.021448 


3 




0.035356 


0.0088164 


8 


6 


0.063622 


0.010494 


12 




0.053932 


0.017486 


13 




0.044184 


0.025712 


3 




0.13959 


0.00015771 


6 


Lio 


0.077434 


0.0031382 


7 


11 


0.034219 


0.014671 


10 


12 


0.064124 


0.0060382 


5 


13 


0.023538 


0.011591 


1 


C 


-0.17033 




2 




0.0078933 


0.0056242 


3 




-0.0028386 


0.026072 



96.4% slgn-te St -based confidence Interval for median overall 
instruction effect: ( 0.0078933 , 0.065415 ) 

95S r-cottf idence intervals far overall instruction effect: 

a. using all 15 instructors (-0.0081273 » 0.065415) 

b. omitting case 14 ( 0.020770 , 0.064942) 

c. omitting cases 9 and 14 ( 0.018664 » 0.052166) 

d. omitting the 5 cases with 3 or fewer subjects 

( 0.022680 , 0.061318) 

^Bracketed classes are from .the some school. 



Hmm ao4 Virlantis for Pt-PR $«or«A for Hales »d Pevalefl In £icporlnMntil ClA&ftea. 
toftotlwr with Catlutttd Virlaotdi of Thoso Mean Dlfforeotes 



HOES 
Hoan Variant e 



FEMALES 
Naau Varlonte 



DIFFERENCE* 



V (Dlff)«* 



1 

.3 



C 



C 

.10 
11 
12 
13 

Ll5 



0.012581 0.0018412 
-0.097411 

'0.055680 0,013357 
0.058839 

0.123610 

0.054343 0.013361 
0.070853 0.010334 

0.138200 

0.077260 0.0028801 
0.12560 0.0094795 
0.12071 0.0099414 
-0.097219 

-0.17033 



0,011587 0.011957 

0.140750 0.011042 

0.023614 0.168060 

0.055052 0.011556 

0.053727 0.021751 

0.013070 0.047151 

0.140280 0.0003125 

0.077781 0.0073003 

-0.00^3343 0.013791 

0.039874 0.0034565 

0.053727 0.0093781 



-o.ioaooo 

-0.196430 
0.035225 

0.06656Z 

0.0006168 
0.057784 

-0.002083 
-0.00052090 

0.12794 

0.080835 

15095 



0.0074377 



0.0060591 
0.0093347 

0.0043702 
0.0074980 
0.0038076 



Ll7 



0.060922 

-0.066065 0.028158 



-0.045136 



0.10606 



97.7% algn-Ce8C*baaad tonfld«nte Interval for nadlan vax dlfforonta la 
ovarail Inatructlon affect: (-0.15095 , 0.080835 ) 

95X t->confX4«Thce Incarval for sen dlfferenca in overall Inatructlon effect: 

(^.080861 , 0.054454 ) 

Hale «eaii mlnua fonala ftean. 



Eatlutefl variance of toeoTh dlfferante. 
J^J^I^K^d claaaea are fro* Cbe aamo achool. 



24 
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Table 8 

Average Scores Differences (PX-PEt) for Experljnental 
Subjects Classified by Ethnic Group. 



CLASS 


N 


WHITE 

Mean 


N 


BUVCK 

Mean 


CHICANO 
» Mft«n 


1 






A 








3 


4 








1 


-.097411 


4 


2 


.072477 


s 








5 


3 


035356 










6 


2 








6 


.088102 


7 


2 








7 


.023505 


a 


a 

Q 


niA1 71 






5 


.060204 


9 














10 


4 


. .069448 


1 


.14653 


u 1 


.040281 


11 


7 


.034219 










12 






10 


.064124 






13 


3 


.099939 


- 2 


-.091065 






14 






1 


-.17033 






16 


1 


.060922 


1 


-.045136 






17 


3 


-.002839 
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the GR£ and its uses in order to reduce any anxiety they may have had 
about it snd 4 sessions of mathematics instruction. In the face of sn 
overal}. effect* we next wish to determine if that effect is because of 
something beyond that attainable from the anxiety reduction session. 
All expftrlmftntal subjects on whom both posttest (FT) and anxiety reduc- 
tion (AK) test scores were available were used to answer this question. 
FT minus AK teat score differences were computed for each class on 
which this combination of scores was available for one or more subjects. 
The results are presented in Table 9* 

For the 11 classes represented in Table 9 the average differences 
batveen the instructional program posttest and the test following 
anxiety reduction were positive in 6 and negative in 5. Ho reliable 
differences due to the A-session instructional program alone were 
detected; the sign-test based 96*4% confidence intervals and the various 
95% ^-*conf idence intervals always include zero* The breakdown for sex 
and ethnic background differences slso failed to show differential 
effects* and those data are not presented here* 

Question 3 

Ws3_the overall effect du^ to the anxiety reduction session slone? 

Given that there was a significant overall effect that does not 
appear to be attributable only to the 4 sessions of mathematics instruc- 
tion» the next question is whether the effect is due to the anxiety 
reduction session alone* In order to investigate this hypothesis data 
from control subjects who had both anxiety reduction (AA) and posttest 
(FT) scores were used. 

The difference AR'FT for the control Ss represents the change in 
score after the anxiety reduction session where the FT^ is a second test 
and ^ a third test. If this difference were positive, it would suggest 
that the anxiety reduction session had a positive effect over and above 
a mere retest (note from Table 4 that in this case FT is a second and 
the AR a third test)* The results of this analysis are presented in 
Table 10* There were 12 control classes in which one or more students 
furnished the desired combination of scores* In two classes the differences 
were negative and in the other 10 the differences were positive* The 
confidence intervals (both sign-test and t) shown at the bottom of the 
table exclude 2ero in all cases* suggesting that the anxiety reduction 
session did have an effect on scores that was not.merely a "second-test" 
effect* Also it is not likely that this effect could be due to a third 
testing since no such effect was found for the experimental subjects even 
after substantial mathematics instruction. Again* breakdowns by sex 
and ethnic background failed to show differential effects* and those data 
sre not presented. 



Table 9 

Means and Variances of Incrasubjecc Differences (PT-AR) 



In Experlaencal Classes 


N CUss 




Variance 


U 


1 


-0.10913 


0.0071867 




* 




1 




^ J 

rL 


0.020382 


0.023286 


•* 




-0.083176 


0.0050465 










1 


- y 


-0.047036 


0.030X28 




> O 

y 


0.088094 


0.00097656 


6 1 


-10 


0.016219 


0.033309 


7 


11 


-0.011415 


0.011321 


6 


12 


-0.054445 


0.065852 


5 


13 


0.044063 


0.0X2425 




*14 














1 


"16 


0.190515 




3 


-17 


0.063901 


0.019373 



93.4% slgn-^test-based confidence Interval -for median increase in 
scores due to coaching* ( -0.054445 > 0.063901 ) 

95Z t^ confidence intervals for increase in scores due to coaching: 

a. using all 11 instructors ( -0.46918 > 0.068367 ) 

b. omitting case 16 ( *0. 053779 > 0.039270 ) 

omitting the 3 cases with 3 or fever subjects 

( -0.073265 , 0.017130 ) 

Bracketed classes are from the same school^ 



-23- 



ERIC 



Table 10 

Means and Variances of Intrasubject Differences (AR-PT) 

for Control Classes. 



5 
7 

15 

8 
2 

16 
11 
4 
2 

7 

10 



Class 



Mean 



0.0029531 



[ 



9 

10 
11 
12 
13 
4 
5 



C 



-0.0065627 
-0.010014 

0.036031 

0.10402 

0.093860 

0.057243 

0.06i373 

0.096633 

0.078235 

0.089094 
0.040070 



Variance 



0.015338 



0.0016835 
0.014715 

0.013633 

0.033342 

0.000019530 

0.014732 

0.024786 

0.010929 

0.017578 

0.024264 
0.013054 



96. IX sign-tesc-based confidence Interval for median Increase In 
scores due to anxiety reduction: ( 0.0029531 , 0.093860 ) 

95% ^-confidence Intervals for Increase In scores due to anxiety 
reduction: 



a. using all 12 Instructors 



( 0.027396 , 0.079759 ) 



b. omitting the 2 cases with 3 or fewer subjects 

( 0.016900 , 0.077268 ) 

^Bracketed classes are from the same school. 



Dla cuss Ion 

Given these resulcs» two points deserve special attention. First* 
the analyses described above address only the question of whether certsin 
differences are significant in a statistical sense* and do not allow for 
an assesenaent of the magnitude of those differences* The results suggest 
that a positive effect that is due to the one session devoted to anxiety 
reduction is found consistently across several classrooma» but that there 
is no conaistent positive effect resulting only from 4 weeks of substantive 
instruction* The absence of a significant effect from 4 weeks of substan- 
tive instruction presents less difficulty for interpretation than does 
the presence. of a positive effect from one session on anxiecy reduction or 
for the combination of anxiety reduction and substantive instruction. 
Hov large a difference is meaningful In terms of the purposes of the 
GKE'-Q? An average increase of, say» 50 scaled*-score points caused by 
instruction might suggest thst there is cause for concern, whereas 
while a statistically significant average increase o£» say* 10 scaled- 
score points might cause no alarmt 

In order to estimate the magnitude of the score increases due to 
the overall program of instruction» pretest (PR) and posttest (PX) scores 
for 88^ experimental subjects in 10 schools were transformed to T-scores 
and compared. The standard scores were computed by setting the mean and 
standard deviation of the pretest corrected-score^ for the entire subject 
pool (N ■ 25A in 10 schools) on whom both PR and PT scores were available 
to 50 and 10 respectively. These parameters were then used to scale the 
corrected score means for the 10 schools to the PR scale. The results of 
the scaling are shown in Table 11* Looking at the ''difference^' coltxmn 
of Table 11 for the 10 experimental groups* we can see that in most 
cases the gains from PR to PT were not substantial* The largest gain was 
+ *598 T-score units for school 7. The sverage gain for the 88 experi- 
mental subjects who completed the program was + t266 T*8core units* The 
reliability of the pre- and posttests is unknown, but if we assume them 
to be between .8 and t9, we would estimate the standard error of a score 
to be between .32 and *45 T-score units* In most cases » then» the score 
gains would be less than the standard error of measurement. 

The population to which these results might generalize is also 
cause for caution* Note from Table 5 that only about 37X of the experi- 
mental group persisted to the posttest. It is highly probable that these 
highly motivated (perhaps test anxious) people differ^rom persons who 
would volunteer for a coaching program and from GRE candidates in 
general . 



The single subject in classroom No. 14 (see Table 6) was omitted from 
these computations. 

^Corrected score * (number correct) (number incorrect) s where 
k • the number of response options for an item. 
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Tablt 11 

N^«# KAocift^ ^H^tl $t««td>rd Deviations of Stflndiirdlzed Tretevt^ ToHttw^tt nnd 
Difference Score* for Cxperlnentel ^nd Control Croups li> 10 SchooU / 



aenoni 


Croup 


u 


S?Handar<Ii«ed Seoree 


Pr^teet \ JPocttest Dlf fiarente^ ' 










01 


I 

C 


13 


^2 . 79 


5.54 


43.11 . 6.20 
47.37 1 7.11 


.42 
2.40 


03 


c 
c 


5 
18 


51.56 
52.23 


9.56 

10. n 


51.09 j 6.24 
52.37 1 10.20 


.47 
.14 




c 
c 


10 
11 


Si. 36 
55.25 


9.29 
7.SH) 


54. $7 
55.50 


8.14 
6.57 


3.51 
.25 




c 
c 


8 

7 


50.4^ 


6.r7 
12.76 


46.72 


6.87 
U.39 


3.66 
-3.74 




c 
c 


25 
29 


46.56 
51.85 


U.35 
10.57 


51.51 12.54 
1 

51. n 1 11:71 


2.95 
-.14 


07 


c 
c 


9 
18 


50.61 

&d.3^> 


8.41 
10.54 


56.5? S.41 
' 49.31 , 9,5i 


5.98 
.97 


08 


c 

c 


7 
19 


48. :3 j 4,35 
54.01 ; 7,99 


50.52 1 4.68 1 2.29 
54.11 ; 10.37 1 .10 


09 


c 
c 


10 
17 


42.0^ 
40.27 


7.78 
6.56 


45.21 ! 9.47 j 3.17 
i 40.46 \ 6.21 \ ,19 


10 


c 
c 


5 
6 


61.16 
58.83 


12.44 
6.48 


$0,95 i 12, fl -.21 
57.65 J 10.05 -I.IS ■ 


12 


c 
c 


5 
18 


59. 3^ j 12.99 
50.71 1 H.06 


58,71 1 U-U 1 -.61 
49.88 1 9.2B \ -.63 

1 h 


Tot a 


c 
c 


88 

1 166 


49. ?^ 1 10.03 
5U.3^ ; 9.86 


52.01 i 10. 5C j 2.63 
50.39 j 10.04 1 ,04 



1. Difference *• incun pretest et/^t)J^rdl«ed seoiv icc^n xtoattesi et^ndnrdlzed 
seore. 

2. The total eub k'i:! pool (H * I?^4) pretext rjtn^^ end ^tondarH tie\-l3tlo:x tjorr 
«et e^tiAl to 50 and 10,. rcn}:(."^;tlyvly. Tlic^v pftratn^tere vtrre then u»ed 
ceale the l^rc^uP rac^e and i^tonL^rd duvljtlons to pratent unlwsi. 



Tz^teet c^^rrttrtt-:*! scoro • 11.55 

Frcteet eorrtjjtod seore stv.:tJatc devlatlr^ii ■ 



.41 



T • 



11.55) 



7..-.1 



+ 50 



\0 
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. Follov-up Survey of SCudenCs 

In November l973f a follow-up survey of scudenca who Initially 
registered for the course was conducted to determine why the attrition 
rate had been so high. Letters w^re sent to all students for whom we 
had addresses. Enclosed with the letter was a postcard on which the 
student was to Indicate whether he or she was still attending the 
Sttsslons andf If nott how many sessions had been attended and why 
attendance had stopped. Additional canftents about the coutse were 
also solicited. Out of the total of 494 letters that were sentt 40 
were returned as undellverable. Of the 220 students (48.5%) who 
responded* 137 (62.3%) Indicated that they were still attending the 
sesslonst snd 83 (37.7% of the respondents) Indicated that they were 
no longer sttendlng the sessions. 

Of those who reported that they were no longer attending the 
sesslonsf 49.5% had attended two or fewer sessions (25.3% not 
any sesslonst 12.1% only one seaslont and 12.1% two sessions) t 
21.7% had attended threet and 9.6% had attended four sessions. 
Only 14.4% had attended five or six sesslonst and 4.8% did not 
Indlcate^how many sessions they had attended. 

The most common reason (53%) cited for dropping out of the 
course was schedule conflicts. Approximately 14.:3% of the respond^ 
ents felt they did not need the revlewt and an equal number stopped 
attending after taking the GR£ In October. About 10.8% changed 
their plans about going to graduate school. 

In addition to giving these reasons for dropping out of the 
courset some students made one or more specific comments about 
the courset as follows: the course was too difficult (12.1%)t 
there were too many tests (9.6%)t and the instructor was not very 
good (4.8%). On the wholet the percentage of positive comments 
about the course (31.3%) was higher than the negative comments 
(13.3%). The additional 55.6% of the students either made no 
comment t or their comments were not classifiable as positive or 
negative. 

Of the 137 students who were still attending class sessions 
82% supplied one or more comments and/or criticisms of the course. 
In general* their comments about the course were favorablet with 
60.6% saying something good about It and only 4.4% saying something, 
negative. About 17.5% of the respondents made no commentst and 
17.5f£ of the comments were unclasslf iable. Among the comments 
and <j^ltlclsma of the course were the following: 
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10*2S thought chat the course had increased their 

confidence in taking the GRE and/or had reduced 
their anxiety about it: 

9.02 felt that there was too much testing; 

8. OX felt that there should have been more or longer 
classes because of the quantity of toaterial to 
be covered; 

9*02 felt that scheduling of the classes was poor; 

5*1% felt that coaching for ch6 verbal section of 
the GRE should be included; 

felt chat the course was too difficult; 

2*2% felt chat the course was coo simple; 

1*5% felc chere was noc enough class work; 

1*5% felc chere should have been more individualized 
insCrucc;ion; 

8«0% made unfavorable comments abouC inscrucCor; 
7«3% made favorable comments abouC inscniccor* 



Follow-up Survey of Campus CoordinaCors and InsCrucCors 

In addition Co che mail survey of studencs» che campus co- 
ordinaCors ^nd instrucCors were asked Co reporC Cheir percepCions 
of Che sCrengChs and weaknesses of che GRZ-Q Special InsCrucCion 
Program* All persons were enthusiascic about the program and 
believed ChaC sCudents beneficed from Che insCrucCion and Che 
Cescingt buC mosc were able Co poinc Co some specific problems 
relaced to Che CesCing macerials and inscruccional sessions* 

Tesclng * The conafensus of Che insCrucCors and coordinaCors 
was ChaC there was Coo much CesCing » especially in the early 
sessions and chaC many sCudenCs were discouraged immediacely by 
che number of cescs and quescionnaires* They also ciCed as 
problems che lack of InmedlaCe feedback on che CesCs and an 
apparenc lack of communi<;ation abouC che accual scrucCure of 
the course* They reporCed ChaC some sCudents ChoughC ChaC every 
session would be a Casting session and Chac there would noC be 
any concenc inscruction* 
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Materlals * In general^ the coordinators and instructofs 
reported that student response to the course materials was over- 
whelmingly favorable* The lnmedlate reinforcement of seeing the 
right answer and a detailed solution of the problems made the 
vorkbooks appealing* The explanations were considered to be 
concise and to the point* and the workbooks presented a good mix of 
all types of problems* Some reported that an outline of the sub- 
ject matter of aach session would be a useful addition to the mlnl- 
lessons* 

Instructional Sessions * Most people felt that the course should 
have been extended In order to cover the material adequately* At 
some schools the Instructors reported that the two-hour class at 
times stretched into a three- or four-hour class* They thought that 
two hours was a good length for a class of this nature* but that 
there should have been more classes^ either two per week for the 
eight-week period or one a week for more than eight weeks* 

The varying level of mathematical sophistication among the 
students in most classes was frequently mentioned by instructors* 
The students^ recent experience with mathematics ranged from **I 
haven^t had any math since high school'* to "I just finished a 
course in advanced calculus (or in statistics)*" TKe more sophis- 
ticated students tended to drop out of the course* ^ereas the 
poorly prepared students were often lost and not able to follow 
the workbook explanations* Many instructors suggested that more 
than one course be developed so that students at different levels 
could be placed either in a brief review course^ a test-taking 
strategies course* or an intensive basic mathematics course. 



-29- 



Sumnary and Conclusions 



A special course was designed to aid volunteer subjects in 
preparing for the CRE'-Q* The course included a short one-session 
discussion (Anxiety Reduction Session) of the GR£ and its uses* 
designed to familiarize candidates yrith the test and thus reduce their 
anxiety* Also included were four sessions devoted to specific instruc- 
tion in the basic mathematics required for the test and strategies for 
approaching the various types of questions « The course was offered 
to volunteers at 12 colleges « Colleges were selected so as to 
maximize the participation of black and Chicano students « 

Initial response (as indicated by the number of students who 
volunteered for the program) was good at all schools* However* 
a ttritiotr throughout the program was high» producing spotty data with 
which to evaluate the effectiveness of the course* Because of the 
spotty nature of the data» the original data analysis plans were 
abandoned and an approach to analysis that used all available data 
was employed* These analyses did not involve direct comparisons of 
experimental group gains with those obtained by comparable control 
groups* Three major questions were investigated: 

1* Was the overall program of instruction helpful in increasing 
GRE-Q scores of the subjects? , 

Z* Was the effect of the program due to the 4 sessions of 
mathraatics instruction alone? 

3* Was the effect of the program due to the single anxiety 
reduction session alone? 

In addition* an attempt was made to determine if there were any 
differential effects that were due to the sex and ethnic backgrounds of 
the subjects* 

The reader is cautioned that differences observed in these analyses* 
while statistically significant » may be due to factors not under the 
control of the investigator such as differential attrition rates and 
levels of motivat^ion for various subject groups* The results c£ the 
analyses suggested: . 

1* there was a small» consistent increase in GRE-Q scores due to 
the overall program; 

Z* the increase appeared to occur early in the program; and 

3* there was no evidence that the program was differentially 
effective for the sexes or the ethnic groups studied* 
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